Generative Models for Learning from Crowds

A large amount of labeled data is required for supervised learning. However, labeling by domain experts is expensive and timeconsuming. A low cost and high e ciency way to obtain large training datasets is to aggregate noisy labels collected from nonprofessional crowds. Prior works have proposed confusion matrices to evaluate the reliability of workers. In this paper, we redene the structure of the confusionmatrices and propose generative probabilistic models which utilize item di culty in label aggregation. We assume that labels are generated by a probability distribution over confusion matrices, item di culties, labels and true labels. We use Markov chain Monte Carlo method to perform the parameter estimation. We also derive a novel variational inference algorithm to perform the posterior inference. To avoid bad local optima, we design a method to preliminarily predict the true label and di culty of each item and initialize the model parameters. Empirical results show that our methods consistently outperform state-of-the-art methods.